Automatic testing of software with structurally complex inputs

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.Includes bibliographical references (p. 123-132).Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties of structurally complex test inputs; presents a technique that automates generation of such inputs; describes the Korat tool that implements this technique for Java; and evaluates the effectiveness of Korat in testing a set of data-structure implementations. Our approach allows the developer to describe the properties of valid test inputs using a familiar implementation language such as Java. Specifically, the user provides an imperative predicate--a piece of code that returns a truth value--that returns true if the input satisfies the required property and false otherwise. Korat implements our technique for solving imperative predicates: given a predicate and a bound on the size of the predicate's inputs, Korat automatically generates the bounded-exhaustive test suite that consists of all inputs, within the given bound, that satisfy the property identified by the predicate. To generate these inputs, Korat systematically searches the bounded input space by executing the predicate on the candidate inputs. Korat does this efficiently by pruning the search based on the predicate's executions and by generating only nonisomorphic inputs. Bounded-exhaustive testing is a methodology for testing the code on all inputs within the given small bound.(cont.) Our experiments on a set of ten linked and array- based data structures show that Korat can efficiently generate bounded-exhaustive test suites from imperative predicates even for very large input spaces. Further, these test suites can achieve high statement, branch, and mutation coverage. The use of our technique for generating structurally complex test inputs also enabled testers in industry to detect faults in real, production-quality applications.by Darko Marinov.Ph.D

Model-based, event-driven programming paradigm for interactive web applications

Applications are increasingly distributed and event-driven. Advances in web frameworks have made it easier to program standalone servers and their clients, but these applications remain hard to write. A model-based programming paradigm is proposed that allows a programmer to represent a distributed application as if it were a simple sequential program, with atomic actions updating a single, shared global state. A runtime environment executes the program on a collection of clients and servers, automatically handling (and hiding from the programmer) complications such as network communication (including server push), serialization, concurrency and races, persistent storage of data, and queuing and coordination of events.National Science Foundation (U.S.) (Grant CCF-1138967)National Science Foundation (U.S.) (Grant CCF-1012759)National Science Foundation (U.S.) (Grant CCF-0746856

Wanted: standards for automatic reproducibility of computational experiments

Those seeking to reproduce a computational experiment often need to manually look at the code to see how to build necessary libraries, configure parameters, find data, and invoke the experiment; it is not automatic. Automatic reproducibility is a more stringent goal, but working towards it would benefit the community. This work discusses a machine-readable language for specifying how to execute a computational experiment. We invite interested stakeholders to discuss this language at https://github.com/charmoniumQ/execution-description .Comment: Submitted to SE4RS'23 Portland, O

Incremental state-space exploration for programs with dynamically allocated data

We present a novel technique that speeds up state-space exploration (SSE) for evolving programs with dynamically allocated data. SSE is the essence of explicit-state model checking and an increasingly popular method for automating test generation. Traditional, non-incremental SSE takes one version of a program and systematically explores the states reachable during the program's executions to nd property violations. Incremental SSE considers several versions that arise during program evolution: reusing the results of SSE for one version can speed up SSE for the next version, since state spaces of consecutive program versions can have sig-ni cant similarities. We have implemented our technique in two model checkers: Java PathFinder and the J-Sim state-space explorer. The experimental results on 24 program evolutions and exploration changes show that for non-initial runs our technique speeds up SSE in 22 cases from 6.43% to 68.62 % (with median of 42.29%) and slows down SSE in only two cases for-4.71 % and-4.81%

Operator-based and random mutant selection: Better together

Abstract—Mutation testing is a powerful methodology for evaluating the quality of a test suite. However, the methodology is also very costly, as the test suite may have to be executed for each mutant. Selective mutation testing is a well-studied technique to reduce this cost by selecting a subset of all mutants, which would otherwise have to be considered in their entirety. Two common approaches are operator-based mutant selection, which only generates mutants using a subset of mutation operators, and random mutant selection, which selects a subset of mutants generated using all mutation operators. While each of the two approaches provides some reduction in the number of mutants to execute, applying either of the two to medium-sized, real-world programs can still generate a huge number of mutants, which makes their execution too expensive. This paper presents eight random sampling strategies defined on top of operator-based mutant selection, and empirically validates that operator-based selection and random selection can be applied in tandem to further reduce the cost of mutation testing. The experimental results show that even sampling only 5 % of mutants generated by operator-based selection can still provide precise mutation testing results, while reducing the average mutation testing time to 6.54 % (i.e., on average less than 5 minutes for this study). I

Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? Without answering such questions, developers, tool builders, and researchers make decisions based on folklore instead of data. In this paper, we use three complementary methods to study in-depth the usage of CI in open-source projects. To understand what CI systems developers use, we analyzed 34,544 open-source projects from GitHub. To understand how developers use CI, we analyzed 1,529,291 builds from the most popular CI system. To understand why projects use or do not use CI, we surveyed 442 developers. With this data, we answered 14 questions related to the usage, cost, and benefits of CI. Among our results, we show evidence that supports the popular claim that CI helps projects release more often. We also discovered that 70% of the most popular projects from GitHub use CI, as well as finding that the overall percentage of projects using CI continues to grow, making it important and timely to focus more research on CI

Influence of sequential fermentation with Lachancea thermotelerans and Torulaspora delbrueckii yeast on chemical composition of Malvasia Istriana wines

Suočavajući se sa sve drastičnijim utjecajem klimatskih čimbenika na kemijski sastav grožđa, enologija traži i proučava nove metode u tehnologiji proizvodnje vina, posebice bijelih, kako bi se očuvale primarne arome te postigla ravnoteža između alkoholne jakosti i ukupne kiselosti. Kao jedno od rješenja nudi se primjena ne- Saccharomyces kvasaca. U ovom istraživanju analiziran je utjecaj sekvencijalne inokulacije komercijalnih sojeva Torulospora delbrueckii i Lachancea thermotolerans sa sojem kvasca Saccharomycem cerevisiae na vino ´Malvazija istarska´. Istraživanje je obuhvatilo inokulacije mošta s ne-Saccharomyces kvascima, a 48 h kasnije i sa sojem S. cerevisae te kontrolnu varijantu isključivo sa S. cerevisae. Ne-Saccharomyces kvasci utjecali su značajno na koncentraciju alkohola, mliječne kiseline te pH vrijednost. Fermentacija sa S. cerevisiae utjecala je na višu koncentraciju ukupnih aromatskih spojeva u vinu. Intenziteti boje i mirisa najbolje su ocijenjeni u kontrolnom uzorku, a metodom redoslijeda najbolje je rangirana ´Malvazija´ iz tretmana T. delbrueckii/ S. cerevisiae.As we face drastic climate change, new methods are being sought in the technology of wine production, especially in production of white wines, in order to preserve the primary aromas and achieve the desired balance of alcohols and acids. As a potential solution the use of non-Saccharomyces yeasts whose metabolism differs from Saccharomyces yeasts is suggested. In this study the influence of sequential inoculation of Torulospora delbriueckii and Lahancea thermotolerans with Saccharomyces cerevisiae on Malvasia istriana wine was studied and analyzed. Malvasia Istriana is the second most widespread white grape variety in Croatia and thus one of the most economically important varieties. The study included inoculations of must with non-Saccharomyces yeasts, and 48 h later with S. cerevisae QA23 strain and fermentation exclusively with S. cerevisae QA23. Non-Saccharomyces yeasts contributed to lower ethanol concentration, especially T. delbrueckii which in co-culture with S. cerevisae was not able to ferment a significant amount of fructose and L. theromotolernas by converting sugar to lactic acid. Although differences within the aromatic profile were observed, none of the non-Saccharomyces strains significantly affected the overall wine aroma. Although the aroma, color and color intensity were best evaluated in the sample produced with the S. cerevisiae strain, the best ranked wine was Malvasia T. delbrueckii/S. cerevisiae

Improved multithreaded unit testing.

ABSTRACT Multithreaded code is notoriously hard to develop and test. A multithreaded test exercises the code under test with two or more threads. Each test execution follows some schedule/interleaving of the multiple threads, and different schedules can give different results. Developers often want to enforce a particular schedule for test execution, and to do so, they use time delays (Thread.sleep in Java). Unfortunately, this approach can produce false positives or negatives, and can result in unnecessarily long testing time. This paper presents IMUnit, a novel approach to specifying and executing schedules for multithreaded tests. We introduce a new language that allows explicit specification of schedules as orderings on events encountered during test execution. We present a tool that automatically instruments the code to control test execution to follow the specified schedule, and a tool that helps developers migrate their legacy, sleep-based tests into event-based tests in IMUnit. The migration tool uses novel techniques for inferring events and schedules from the executions of sleep-based tests. We describe our experience in migrating over 200 tests. The inference techniques have high precision and recall of over 75%, and IMUnit reduces testing time compared to sleepbased tests on average 3.39x

Continuous Integration (CI) Needs and Wishes for Developers of Proprietary Code

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI being one of the most widely used processes in software engineering, we do not know what motivates developers to use CI, and what barriers and unmet needs they face. Without such knowledge developers make easily avoidable errors, managers reduce the productivity of developers by making misinformed decisions, tool builders invest in the wrong direction, and researchers miss many opportunities for improving the software engineering practice. Given the large fraction of proprietary code development, understanding how proprietary developers are using CI is vital to improving it. We present the first study of how CI is used in the proprietary development of software. We conduct 16 semi-structured interviews with developers from different industries and development scale. We generalize these findings by surveying 523 developers from all over the world. We find that 78% of developers feel more productive when using CI, and 85% think that using CI causes developers to give more value to automated testing. However, unlike open-source developers, many proprietary developers want to use CI but are not allowed to do so. 50% of developers have problems troubleshooting CI builds, and more than half of developers want easier configurations for CI tools and services